Carnegie  Mellon 

Software  Engineering  Institute 


Pittsburgh,  PA  15213-3890 

Transformation  of  a  Software 
Development  Organization  Using 
Software  Acquisition  Principles: 
A  Case  Study 


DFSG/PN: 

H.  Borst,  F.  Somali, 
S.  Fritts,  L.  Hamlton 


SEI: 

P.  Obeirxtorf, 
E.  VUubel 


SSTC  1  May  2006 


Sponsored  by  the  U.S.  Department  of  Defense 
©  2006  by  Carnegie  Mellon  University 


page  1 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

MAY  2006  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2006  to  00-00-2006 

4.  TITLE  AND  SUBTITLE 

Transformation  of  a  Software  Development  Organization  Using  Software 
Acquisition  Principles:  A  Case  Study 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon  University, Software  Engineering  Institute, 5000  Forbes 
Avenue, Pittsburgh, PA, 15213-3890 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARSTRATT 

1 8 .  NUMBER  1 9a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  aS 

unclassified  unclassified  unclassified  Report  (SAR) 

23 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Carnegie  Mellon 

Software  Engineering  Institute 

Agenda 

•  Background 

•  Outline  of  SEI  Study  Results 

•  The  Transformation 

•  Results 


©  2006  by  Carnegie  Mellon  University 


page  2 


Carnegie  Mellon 

Software  Engineering  Institute 

Background 

•  Late  2001 :  Air  Force  leadership  requested  that  the  SEI 
conduct  a  brief  probe  to  investigate  software  quality  problems 
with  the  newly  released  Military  Personnel  Data  System 
(MilPDS). 

-  many  airmen  experienced  pay  problems 

-  personnelists  complained  of  poorly  functioning  software 
with  a  constant  flow  of  patches/fixes 

•  Development,  fielding,  and  sustainment  of  MilPDS  was  owned 
by  an  office  within  the  Air  Force  Personnel  Center  (AFPC). 

-  no  acquisition/programmatic  oversight  or  true  Program 
Management 

-  development  budget/resources  “taken  out  of  hide” 

-  indistinct  line  between  “customer”  and  “developer” 

•  Late  2002:  the  SEI  conducted  a  six-week,  intensive  study 

•  2004:  the  SEI  returned  to  conduct  a  follow-on  study 
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SEI  Study  Results1 


•  Requirements 

-  requirement  to  “make  it  look  like  legacy” 

-  no  clear  requirements  management  process 

-  no  distinction/differentiation  between  defects  and  new 
requirements/enhancements 

-  advantages  of  powerful  ERP  systems  not  recognized  or  used 

•  Data 

-  data  irregularities  handled  on  case-by-case  basis,  with  little 
effort  to  identify  larger  patterns  or  root  causes 

-  data  migrated  from  legacy  system  was  not  clean,  causing 
problems  in  implementation  of  MilPDS 

•  Engineering  Processes 

-  no  one  owned  software  development  process 

-  multiple  teams  used  multiple  processes;  in  some  cases, 
competing  processes  existed 

-  gaps  in  process,  no  process  documentation 
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SEI  Study  Results2 


•  Deployment/Support  Processes 

-  limited/incomplete  testing,  focused  largely  on  “happy  path” 

-  typical  testers  not  qualified/experienced 

-  little  to  no  CM  -  code  deployed  without  controls;  constantly 
issuing  emergency  patches,  which  frustrated  customers 
and  introduced  new  defects 

•  Products 

-  heavily  customized  COTS  software  implementation 
(modified  source  code) 

-  3M+  SLOC,  with  little/no  documentation  (user  manuals, 
design,  code  standards,  etc.) 

-  database  platform  approaching  obsolescence,  hampering 
supportability/maintainability 

-  relationship  with  COTS  vendor  not  actively  maintained 
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SEI  Study  Results3 


•  Personnel  &  Management 

-  not  enough  personnel  with  the  appropriate 
skills/training  in  development,  test,  etc. 

-  majority  of  personnel  “blue-suiters”  who  would  soon 
rotate  out;  combined  with  lack  of  documentation,  led 
to  supportability  problems 


•  Acquisition 

-  funding  taken  “out  of  hide” 

-  programmed  funding  for  future  development/ 
sustainment  was  not  evident 
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Key  SEI  Recommendations:  2001 

Reinstate 

full  program  management 
a  technical  lead/system  architect  with  authority 

Secure  a  long-term  funding  line 

for  continued  development 
for  technology  refresh 
for  sustainment 

Consider  the  organizational  implications 

Maintaining  and  evolving  a  COTS-based  system  is  very 
different  from  the  “old  way.” 

Old  concepts  of  “maintenance”  must  be  replaced  by  a  new 
mindset  of  operation  and  evolution. 

There  will  be  major  new  releases  for  the  life  of  the  system. 
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Status  of  Key  SEI 
Recommendations:  2002 


Recommendation 

Oct  02 
Status* 

System/  Process 

Cease  proliferation  of  releases 

+ 

System/  Process 

Define  and  use  software  processes 

+ 

Mgt/  Training/  Documentation 

Conduct  personnel  skills  assessment 

+ 

System/ SW/  Test 

Base  tests  on  requirements,  mature 
processes,  aggressively  look  for  failures 

+ 

Documentation 

Develop  technical/functional 
documentation 

+ 

Data  Cleansing 

Validate  data  fields  in  MilPDS 

+ 

*Excerpted  from  internal  status  briefing  chart,  October  2002. 
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Transformation 


•  Reorganize 

•  Focus  on  repeatable  development  process  with  clear 
definition  of  stakeholder  responsibilities 

•  Institute  measurement  program 

•  Implement  requirements  prioritization  process 

•  Make  changes  for  acquisition 

•  Create  an  acquisition  strategy/dual  responsibility 
strategy 
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Previous  Organization 


Pre-System  Program  Office  (SPO): 

•  Air  Force  Personnel  Center  (AFPC),  Directorate  of  Personnel 
Data  Systems  (DPD) 

-  Responsible  for  development,  maintenance,  network 
operations,  program  management,  security,  system 
administration,  architecture,  engineering,  database 
management,  etc. 

•  AFPC,  Directorate  of  Personnel  Support  (DPS) 

-  matrixed  to  AFPC/DPD 

-  Responsible  for  providing  the  functional  requirement, 
operational  test  and  evaluation,  acceptance  testing, 
documentation  (Help  screens) 

•  Activity  Development  Teams  consisting  of 

-  functional 

-  developer 

-  tester 
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Reorganization 


SPO 


•  Performing  true  Acquisition  Program  Management 

•  Contracting 

•  Financial  Management 

•  Development/Programming 

•  Engineering 

-  Database  Administration/Management 

-  Technical  Requirement  Analysis 

-  System/Integration  Testing 

-  Configuration  Management 

-  Quality  Assurance 
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Repeatable  Development  Processes 


•  Organizational  goal  (“CMMI  Level  2  in  2”) 

•  Re-chartered  AFPC  SEPG  to  SPO  SEPG 


-  Narrowed  scpoe  from  improving  AFPC  business  processes 
to  defining  MilPDS  system  maintenance  processes 

•  Chartered  Process  Action  Teams  (PATs) 

-  PAT  performance  was  unsatisfactory 

-  Placed  functional  managers  as  process  owners-instant 
accountability 

•  Practitioners  trained  on  new  development  processes 

•  Implemented  QA  audits  on  all  MilPDS  releases;  identified  non- 
compliance  issues 

•  Performed  series  of  SCAMPI  appraisals  to  verify  CMMI 
compliance 

-  SCAMPI-C  (Mar  05) 

-  SCAMPI-B  (July  05) 

-  SCAMPI-A  (Nov  05) 


MilPDS  Appraised  at  CMMI  Level  2 
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Measurement  Program 


•  Established  strategic  goals  at  Leadership  Summit  Fall  2002 

-  stabilize  MilPDS 

-  develop  a  quality  team 

-  posture  for  the  future 

•  Implemented  SEI-supported  Goal-Driven  Software 
Measurement 

-  process  compliance 

-  resources  and  cost 

-  product  quality 

-  process  performance 

•  Measurements  used  for  stabilization  and  performance 

-  prepare  for  CMMI  Level  2  SCAMPI 

•  Measurements  scope  expanded  to  other  projects 
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MilPDS  Pay  Defects  &  Dataloads  per  FY  (2001-2006) 
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Requirements  Prioritization1 


•  Customer  has  a  responsibility  to  know  their  business,  to 
communicate  their  needs,  and  to  make  tradeoffs 

-  requirements  liaison  in  place  to  “translate”  customer  needs 

-  constant  negotiation 


•  Facilitates  expectation  management  and  setting  with 
customer/user  community 

-  Requirements  Management  Board  briefed  quarterly 

-  SPO  provides  customer  with  status  refresh  daily 

•  Customer  is  responsible  for  ensuring  that  the  need  is  reflected 
in  priority  order 

-  fixed  number  of  resources 

-  continual  policy  changes  in  AF 

-  continual  technological  advances  to  take  advantage  of 
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Requirements  Prioritization2 


Not  everything  can  be  Priority  1 

•  Customer  participation 

-  Requirements  Management  Board  (RMB)  process 

-  “rack  &  stack” 

-  continual  negotiation 

•  SPO  process 

-  continual  “churn”  of  analysis/programming 

-  static  and  aggressive  testing  windows 

-  configuration  CONTROL 

-  process  assurance  “cops” 

•  System  Configuration  Control  Board 

-  chartered  to  make  decisions 

-  approves  baseline  to  all  releases 

-  uses  risk  management  process  to  approve  out-of-cycle 
requests 
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Changes  for  Acquisition 

SPO  Stand-up 

Focus:  Fix  MilPDS 

•  Absorbed  analysis/programming  staff 

•  Hired  experienced/qualified  Acquisition  Program  Managers 

-  Absorbed  program  management  staff 

•  Hired  experienced/qualified  Engineering  Staff 

-  Built  a  testing  staff  and  implemented  aggressive  test 
program 

•  Hired  experienced/qualified  contracting  officers 

•  Hired  experienced/qualified  financial  managers 

•  Provided  needed  training  (CMMI/SEI) 


©  2006  by  Carnegie  Mellon  University 


page 


Carnegie  Mellon 

Software  Engineering  Institute 


Acquisition  Strategy 

•  Consolidation  of  contracts 


•  Aggressive  contracting  practices 

-  correcting  contracts  awarded  prior  to  SPO  stand-up 

-  following  contracting  processes  for  all  future 
acquisitions 
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Results 

SCAMPI  appraisal  in  Nov  05 

Program  Director  Goal  of  CMMI  Level  2  in  2  years 


2005  Shiely  Award  Winner  -  Best  Program  Office  @  ESC 

•  “Personnel  systems  problems  evaporated,  exceeded 
expectations.. .off  CSAF  Top  6” 

•  “Standardized  requirements  process,  implemented  integrated 
requirements  toolset,  ensured  user  priorities  met” 

•  “222%  reduction  in  new  defects-69%  reduction  in  total 
defects-achieved  during  33%  turnover  in  workforce” 

•  “Consistently  used  a  Systems  Configuration  Control  Board-a 
proven  technical  advisory  for  all  system  changes” 

•  “Improved  functional  office  review  process  and  configuration 
control  process-impact:  higher  quality  analysis” 

•  “Transformed  strategy;  awarded  1st  unit  performance-based 
contract-now  at  80%,  exceeding  40%  OMB  goal” 

•  “ID’d  technology  ‘quick-wins’  to  reduce  customer  workload-80% 
implemented  immediately-now  a  ‘big  win’” 
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Key  SEI  Recommendations 


Recommendation 

Oct  02 
Status 

Apr  05 
Status* 

Apr  05  Comments* 

System/ Process 

Cease  proliferation  of  releases 

+ 

Release  frequency  changed;  allows  for  more 
comprehensive  testing 

System/  Process 

Define  and  use  software  processes 

4 

f 

Change  Management  Process  documented, 

SEPG  leading  CMMI  Level  2  efforts 

Mgt/  Training/  Documentation 

Conduct  personnel  skills  assessment 

4 

+ 

Positional  skill  assessment  complete;  training 
program  in  development 

System/  5 W/  Test 

Base  tests  on  requirements,  mature 
processes,  aggressively  look  for  failures 

4 

+ 

Test  process  being  scrubbed, 56%  complete; 
updating/reviewing  test  processes;  Rqmts,  test 
cases,  code  will  be  linked  with  new  tool  (Oct- 
Nov  04) 

Documentation 

Develop  technical/functional 
documentation 

4 

Documentation  of  system  requirements,  code 
and  test  cases  ongoing;  sys  rqmts  40%,  code 
documentation  started 

Data  Cleansing 

Validate  data  fields  in  MilPDS 

4 

-4 

System  supports  data  validation;  DPS  is  POC 
for  actual  data  cleansing 

*Excerpted  from  internal  status  briefing  chart,  April  05. 
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Closing  Comments 


•  It’s  critical  to  have  a  few  “champions”  who  “understand 
and  get  the  job  done”  -  they’ll  show  up  in  surprising 
positions  and  guises 


•  Senior  leadership,  top-down  commitment,  boss  has  to 
say  AND  do;  emphasize  accountability 


•  This  isn’t  an  overnight  change  -  it  didn’t  get  bad 
overnight  and  you’re  not  going  to  change  it  all  in  a  day 

•  Hire  qualified  personnel  and  train  the  ones  who  aren’t 

-  supplement  institutional  knowledge  with  fresh  new 
eyes 
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Contact  Information 


DFSG/PN: 

Frankie  Sorrell 
Chief  Engineer 

marie.sorrell@randolph.af.mil 

Lynne  Hamilton 

Program  Manager 

lynne.hamilton@randolph.af.mil 

Scott  Fritts 
SEPG  Lead 

scott.fritts@randolph.af.mil 


SEI: 

Eileen  Wrubel 

eow@sei.cmu.edu 

Tricia  Oberndorf 

po@sei.cmu.edu 
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